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1 .  INTRODUCTION 

This  report  reflects  the  work  done  during  the  second  year  of 
the  three  year  project  on  "Control,  Synchronization  and  Fault- 
Tolerant  Operations  in  Variable  Topology  Multicomputer 
Systems".  The  first  year  has  been  reported  in  [IT.  The 
overall  purpose  of  the  research  work  has  been  to  investigate 
the  architecture  of  a  network  of  low  cost  computers  (mini-  or 
microcomputers)  linked  with  serial  communication  paths  which 
can  be  reconfigured  according  to  the  needs  of  each  computation. 
This  led  to  a  novel  architecture  named  Variable  Topology 
Multicomputer,  or  VTM  for  short,  developed  by  a  previous 
three  year  US  European  Office  Grant  [2],  [3], 

The  construction  of  parallel  computer  structures  where 
relatively  large  numbers  of  computers,  reaching  two  or  three 
digit  size  has  become  feasible  due  to  the  advances  in  LSI 
technology  which  led  to  very  low  cost  microprocessors, 
memories,  and  ancilliary  circuits.  Minicomputers  have  already 
introduced  some  degree  of  de-centralization  in  real  time 
systems  where  it  has  become  possible  to  place  a  computational 
resource  nearer  to  the  point  where  such  a  requirement  exists. 
Such  loosely  linked  computer  networks  are  particularly 
attractive  where  the  computational  task  is  spatially  distri¬ 
buted  and  has  to  be  performed  in  real  time.  Microcomputers 
have  accelerated  this  tendency  to  a  greater  extent  by  offering 
the  designer  remarkably  flexible,  powerful  and  low  cost 
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components.  Microelectronics  industry  has  produced  more 
powerful  circuits  such  as  16-bit  processors  and  32-bit 
processors  currently  being  introduced.  On  the  high  perfor¬ 
mance  side,  the  US  Defence  Department  project  is  an  effort  to 
achieve  very  high  performance  microprocessors  [4] .  Thus  the 
technology  limitations  for  a  microcomputer  as  a  point  computing 
resource  is  yet  to  be  reached.  The  availability  of  high 
performance  yet  low  cost  microprocessors  is  making  the  use  of 
aggregates  of  such  components  as  a  tightly  coupled  structure 
more  and  more  attractive.  The  software  support  available  for 
more  powerful  microprocessors  has  also  greatly  improved;  in 
addition  to  having  high  level  language  compilers  e.g.  Fortran 
and  Pascal,  sophisticated  operating  systems  such  as  Unix  are 
available  for  these  machines.  This  provides  a  much  richer 
environment  for  developing  multi-microprocessor  systems  [5] . 

Another  significant  development  has  been  the  emergence  of 
Local  Area  Networks  (LAN's).  These  provide  an  interconnection 
medium  for  relatively  large  numbers  of  computers  (mainly  mini 
or  micro).  The  contention  bus  scheme  of  Ethernet  [6],  the 
circulating  token  structure  of  the  Cambridge  Ring  [7]  and 
token  passing  ring  [8]  are  some  of  the  well  known  approaches. 
These  are  fixed  topology  architectures  yet  they  offer  fairly 
standardised  interconnection  media,  very  different  from  the 
store-forward  scheme  used  for  long-haul  networks.  However  the 
standardisation  is  not  being  extended  to  the  communication 
protocol  level  which  goes  beyond  the  link  level  protocols. 
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Many  of  the  underlying  control  and  synchronization  problems 
are  receiving  greater  attention,  some  due  to  LANs,  others 
due  to  the  greater  use  of  distributed  computing  [9] . 
Distributed  Operating  Systems  are  of  much  interest  as  are 
the  higher  level  language  constructs  for  synchronisation  and 
control,  for  example  the  rendezvous  concept  of  the  Ada 
language  is  designed  to  achieve  synchronization  between 
cooperating  tasks. 

These  developments  had  fundamental  implications  in  system 
design.  Rather  than  sharing  one  powerful  central  machine, 
individual  processes  are  identified  and  allocated  to  separate 
smaller  machines  for  execution.  Novel  computer  structures 
have  emerged  for  implementing  real  time  systems  where  the 
computational  resources  are  distributed  and  interacting 
according  to  the  requirements  imposed  by  a  particular 
application.  Currently  there  exists  an  unprecedented  range  of 
components  to  be  able  to  tailor  a  system  such  that  computa¬ 
tional  power  of  required  dosage  is  applied  to  points  where 
data  acquisition  and  processing  functions  are  required.  The 
current  state  of  system  technology,  however,  is  not 
sufficiently  developed  to  provide  a  designer  with  well 
integrated  hardware,  software  and  communication  tools  to 
build  such  networks  as  distributed  computer  systems.  Given 
the  complexity  of  many  distributed  computer  systems,  there 
exist  very  few  design  tools,  if  any,  to  understand  and 
evaluate  many  of  the  options  and  performance  under  varying 
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operating  conditions  of  the  system  prior  to  implementation 
and  commissioning.  Often,  evaluation  is  done  subsequent 
to  installation  as  an  afterthought,  more  for  dealing  with 
bottlenecks  rather  than  to  ensure  that  the  system  functions 
within  well  prescribed  target  specifications. 

!  During  1979-81,  a  distributed  computing  simulation  package 
called  MICROSS  was  designed  and  developed, by  Dr.  Bozyigit 
and  myself,  supported  by  a  Science  and  Engineering  Research 
Council  research  grant.  'The  details  of  this  package  are 
introduced  in  this  report.  This  system  enables  a  designer 
to  enter  the  specifications  of  a  multi-processor  system  in 
terms  of  topology,  transmission  line  and  processor  character¬ 
istics  in  an  interactive  manner.  Following  a  simulation 
phase  the  reporting  is  done  by  plotting  performance  figures 
such  as  delay  times,  throughput,  etc..  Thus  MICROSS  is  a 
powerful  package  to  evaluate  various  architectural  approaches 
and  topologies  graphically  in  an  interactive  manner.  MICROSS 
facilities  are  now  offered  as  a  service  to  various  research 
groups  and  laboratories  who  would  like  to  test  the  design 
options  that  they  may  have  in  distributed  systems.  MICROSS 
is  seen  as  an  important  tool  to  evaluate  a  given  architecture 
against  a  given  application  requirement . 

In  this  report,  Section  1  presents  an  introduction  to  the 
MICROSS  modelling  and  sumulation  package.  Sections  2 
describes  MICROSS  in  a  more  detailed  manner,  in  particular 
the  way  a  distributed  computer  system  is  specified  for 


5 


modelling  purposes.  Section  4  introduces  the  graphical 
aids  provided  by  MICROSS.  Chapter  5  presents  the  outcome  of 
a  number  of  experiments  carried  out  by  H.  English  to  evaluate 
the  influence  of  interconnection  topology  on  performance. 

This  is  done  by  taking  two  sets  of  network  sizes:  the  first 
set  contains  9  node  configurations  and  the  second  set 
contains  16  node  configurations.  Clearly,  facilities  are 
here  to  study  other  network  sizes  and  topologies.  Section  6 
is  the  conclusions. 
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2.  MICROSS  SYSTEM  OVERVIEW 

2.1  Modelling  of  Distributed  Computer  Systems 

There  have  been  numerous  studies  towards  modelling  DCS.  The 
great  majority,  however,  have  been  based  on  theoretical 
approaches  which  require  mathematical  understanding  of  the 
model  in  relation  to  the  particular  system  under  study.  The 
designers  of  DCS,  as  well  as  those  in  other  fields,  are  not 
necessarily  in  command  of  the  mathematics  involved.  This  gap 
can  normally  be  filled  by  specialists  who  are  able  to 
formulate  a  particular  model  and  present  the  results  to  the 
others  involved.  This  technique,  apart  from  the  cost  and  the 
time  factors  involved,  presents  practical  difficulties. 

A  considerable  proportion  of  modelling  studies  is  based  on 
digital  simulation  techniques.  Especially  with  high  availabi¬ 
lity  of  computer  power  this  has  become  a  widespread  practice. 
However,  there  is  not  any  efficient  formal  approach  despite  the 
availability  of  some  general  purpose  simulation  languages  or 
packets  (GPSS,  SIMULA,  GASP,  etc.).  Here  again,  practice  has 
been  the  development  of  dedicated  simulation  systems. 

2.2  MICROSS  approach  of  modelling  of  DCS 

Development  of  MICROSS*  is  based  on  an  attempt  to  formalize 


*  MICROSS'  name  was  originally  used  as  an  abbreviation  of 
multi-MICROprocessor  based  System  Simulation 


7 


the  interaction  of  basic  components  at  functional  level. 

The  use  of  MICROSS,  it  is  hoped,  will  shorten  the  design 
cycle  by  reducing  the  modelling  effort  involved  in  the 
design  of  a  distributed  computer  system.  The  foreseen 
advantages  of  MICROSS  are  as  follows: 

1 .  A  designer  can  incorporate  his  system  into 
MICROSS  readily  since  the  terminology  used 
agrees  with  DCS  terminology  which  he  is 
also  accustomed  to. 

2.  MICROSS  is  interactive.  The  user  is  guided 
through  the  definition  of  his  system 
components . 

3.  Interactive  reporting  facility  provides  easy 
examination  and  evaluation  of  a  particular 
configuration. 

4.  Embedded  standard  networking  alternatives  i.e. 
protocols  and  routing  provide  alternative 
design  options. 

5.  The  graphics  aid  is  provided  at  three  levels. 
Definition  of  the  system  can  be  aided  graphi¬ 
cally.  State  of  simulation  can  be  graphically 
displayed  at  predetermined  points  in  time 
using  incremental  simulation  times.  The 
graphical  presentation  of  the  results  provides 
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for  speedy  decision  making. 

6.  The  adaptability  of  MICROSS  regarding  the 
computer  installation,  is  high,  the  non-graphic 
MICROSS  source  package  is  coded  in  FORTRAN 
language  which  provides  wide  portability.  The 
graphical  MICROSS  uses  GINOF  graphics  package, 

a  PLOTIO  version  will  soon  be  implemented  which 
will  broaden  the  implementation  area  further. 

7.  In  view  of  contemporary  widespread  implementation 
and  use  of  distributed  computer  systems, 
geographically  dispersed  as  well  as  local  computer 
networks,  MICROSS  can  be  used  as  an  educational 
tool  to  provide  appreciation  of  DCS  and  give 
insight  into  their  characteristics. 


2.3  Functional  components  of  a  DCS 

2.3.1  What  is  a  function? 

The  definition  of  a  function  depends  on  the  level  with  which 
the  function  is  associated.  For  example,  at  one  extreme  a  DCS 
consists  of  two  main  functional  components,  nodes  which  under¬ 
take  application  as  well  as  switching  tasks,  and  communication 
lines  which  have  certain  capacity  of  transmission.  At  the 
other  extreme,  there  is  circuit  and/or  instruction  level  of 
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functional  decomposition  which  provides  extensive  details 
and  many  more  options. 

The  highest  level  may  be  considered  too  crude,  due  to  the 
availability  of  various  alternative  arrangements  of  a  node 
made  possible  with  the  advances  in  LSI  technology.  The  lowest 
level,  on  the  other  hand,  contains  too  many  redundant  details 
which  can  easily  distract  the  purpose  of  simulation,  and  may 
even  make  it  unmanageable  or  impossible. 

This  study  towards  generalization  of  DCS  modelling,  has 
assumed  an  intermediate  level  where  redundant  details  are 
excluded  but  enough  means  of  control  can  be  given  to  the 
designer  over  the  hardware  and  software  arrangements  of  each 
node  involved  in  this  system. 

2 . 3 . 2  MICROSS  functional  components  and  specification 
considerat ions 

A  node  in  a  DCS  consists  of  two  main  components  (Figure  2.1): 

1 .  Host  side 

2.  Communication  side  as  shown  in  Figure  2.1 

The  host  side  has  functions  associated  to  the  application  and 
communication  side.  The  communication  side,  on  the  other  hand, 
performs  switching  tasks  which  route  messages  towards  their 
ultimate  destination.  Thus  the  hardware  specifications  that 


<  DC 
LJ  O 


z:  i — i 

z:  o 

°  ^ 

.OJ  CL 


<  DC 
1 — >  O 

:  oo  — 

2:  oo  •—« 

ID  LlJ  ■ — . 

^  ^  00 
r  o  q_ 

OOC  u 
' — i  Cl  i — , 


Figure  2.1  TWO  NODES  AS  SIMULATED  IN  MICROSS 
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are  related  to  the  two  components  would  be  speed  and  memory 
with  which  the  host  and  the  communication  node  perform  their 
functions.  These  speeds  in  MICROSS  are  given  in  kilo  (1000) 
instructions  per  second.  This  relates  to  the  size  of  soft¬ 
ware  involved  in  carrying  out  specific  tasks  involved.  The 
memory  size  is  not  a  factor  of  high  importance  but  it  should 
be  sufficient  for  the  allocation  of  the  software  and  the 
information  in  transit. 

The  local  interface  between  the  host  and  the  local  communi¬ 
cation  processors  is  specified  by  the  protocol  employed  and 
the  transmission  speed  of  the  interface  circuitry.  [The 
transmission  media  in  the  communication  subsystem  is  often 
connected  to  the  communication  mechanism  (or  protocol) 
between  the  communication  nodes.] 

The  interface  transmission  timing  is  modelled  as 

t  »  a  ♦  Vb 

where  a  is  a  fixed  value  per  unit  information,  b  is  a 
coefficient,  and  a  is  the  number  of  units  of  data  to  be 
transmitted.  The  units  depend  on  interpretation  if,  say,  9, 
is  in  bits  and  b  is  in  mega  bits  per  second,  then  a  and  t 
are  in  microseconds. 


3. 


MICROSS  system  details 


3.1  Simulation  technique 

The  simulation  of  DCS,  obviously  involves  concurrent  execution 
of  various  components  on  a  single  processor  i.e.  a  mainframe 
computer  environment.  It  is,  therefore,  important  how  the 
global  time  is  advanced  regarding  discernible  functional  units 
such  as  host,  communication  processors,  I/O  ports,  and 
communication  links. 

MICROSS  employs  an  event  based  discrete  simulation  technique. 
This  technique  involves  the  ordering  of  the  events,  which  are 
referred  to  as  transaction  (TX  in  short) ,  in  time  and  in 
precedence . 

An  event  in  the  system  can  appear  in  one  of  the  five  states: 

1 .  scheduled:  waiting  to  take  place 

2.  active:  taking  place 

3.  suspended  (or  blocked):  waiting  for  the  requested 

facility  to  become  free 

4.  pre-empted:  prevented  by  a  higher  priority  TX 

5.  dormant:  TX  completes  its  life  through  the  system 

and  terminates  for  good. 

Figure  3.1  shows  the  transition  of  TX  states. 
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The  states  1  and  2  have  the  common  characteristic,  scheduled 
to  wait  for  a  certain  duration  of  time  before  competing  for 
next  facility.  The  state  1,  however,  until  the  involved  TX 
enters  the  system,  for  the  first  time,  takes  place  outside  of 
the  system  with  no  effect  on  the  system  at  all. 

A  TX  enters  a  blocked  state  if  the  facility  it  is  competing 
for  is  busy.  A  blocked  TX  resumes  competing  upon  each 
transition.  The  state  4  has  not  yet  been  implemented  in 
MICROSS,  although  provisions  are  made  to  include  it.  The 
state  5  is  obvious.  The  transactions,  be  it  control  or 
data ,  come  to  an  end  at  a  particular  host  or  communication 
node.  This  state  is  referred  to  as  TX  termination  state  later 
in  the  simulation  and  reporting  parts. 

The  facilities  involved  regarding  the  simulation  model  are 
host  computers  and  communication  processors,  and  the  transmission 
medium  with  host-to-communication  subsystem  interface  and 
communication-to-communication  processor  connection. 

3.2  Basic  MICROSS  software  system 

In  view  of  the  general  modelling  aspects  mentioned  in  the 
previous  parts  the  presentation  in  this  part  covers  basic 
descriptions  of  main  compontents  of  the  MICROSS  software 


system. 
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MICROSS  is  organized  in  the  form  of  three  loosely  tied 


submodels 

and  two  utilities 

1  . 

Definition 

2. 

Simulation 

3. 

Reporting 

4. 

Restart 

5. 

System  snapshot. 

The  interface  between  all  five  is  monitored  under  the  control 
of  a  small  MONITOR. 

These  main  constructs  are  also  transparent  to  the  user. 

While  he  is  guided  interactively  through  the  MICROSS  system 
he  will  know  the  submodel  in  charge  of  the  current  action, 
by  the  prompt  carrying  the  name  of  the  submodel.  Each  model 
has  well  defined  interface  with  the  monitor.  There  is  no 
control  from  one  to  another  unless  it  comes  through  the 
monitor . 

In  this  secton  basic  tasks  undertaken  by  each  submodel  are 
introduced  at  a  higher  level.  Further  details  are  included 
in  the  User  Manual. 

3.2.1  Control  of  MICROSS  submodels:  MONITOR 


The  MONITOR  routine  shows  all  the  global  data  structures  with 
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control  over  user  and  other  submodels  of  MICROSS.  Figure  3.2 
shows  the  system  flowchart  of  MONITOR.  With  the  interactive 
control  the  user  can  switch  between  the  submodels  as  many 
times  as  found  necessary  until  a  convincing  architecture  is 
achieved,  judged  by  the  analysis  of  the  performance  results 
of  each  alternative. 

MONITOR  satisfies  the  need  of  clear  user  interface  with 
MICROSS.  The  MONITOR  is  the  only  path  to  enter  a  subsystem 
and  also  the  only  path  to  exit  the  systetn  properly. 


3.2.2  System  definition  and  initialisation 

This  section  provides  all  the  data  necessary  for  the  simulation 
regarding  the  capabilities  of  the  hardware  components  such  as 
host  computers  and  communication  subsystem,  and  applicaton  such 
as  patterns  of  external  message  send  requests  generated  at  the 
hosts.  The  upper  limits  are  put  for  modelling  constructs  such 
as  scheduling  lists,  and  queuing  buffers  at  host  and  communi- 
caton  subsystem.  The  options  regarding  the  communication 
protocols  and  routine  techniques  are  also  specified  in  this 
section. 

The  data  structures  used  to  implement  simulation  constructs 
and  also  statistics  on  the  facilities  are  initiated  in  this 
system,  as  well.  The  system  data  can  be  grouped  into  four 


classes : 
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MONITOR 


Figure  3.2  MONITOR 
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A.  Topology  Definition 

Providing  he  stays  within  limits  imposed  by  the  system, 
a  user  can  either  select  his  topology  from  a  range  of 
'regular'  and  'standard'  topologies  or  define  an 
irregular  network  using  either  interactive  graphical 
input  or  by  means  of  a  connection  table.  An  option 
available  to  interactive  users  is  to  construct  their 
network  with  a  geographical  map  as  background. 

After  definition  the  topology  information  can  be 
stored  in  a  data  file  for  future  use. 

B.  Hardware  Data 

Speed  of  local  processor:  kilo  instructions/sec. 

Speed  of  Communication  processor:  kilo  instructions/sec. 
Speed  of  local  transmission  medium:  kilo  bits/sec. 

Speed  of  the  lines  between  communication  nodes: 
kilo  bits/sec. 

Local  memory  at  the  host  for  communication  software 
only:  kilo  bytes 

Communication  memory  at  the  communication  processor: 
kilo  bytes. 

Output  queue  lengths  in  terms  of  number  of  messages. 

The  processor  speeds  are  used  to  simulate  the  seizure 
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duration  of  the  processors  given  a  task  requiring  a 
certain  number  of  instructions.  Memory  size,  on  the 
other  hand,  is  checked  for  the  sufficient  buffer  space 
and  is  not  a  critical  factor  with  the  low  cost  factor 
involved.  The  lines  delay  the  transactions  by  an 
amount  proportional  to  their  (TX's)  lenuth. 

C.  Routing  Data 

The  routing  technique:  mo-  decidtd  up'>n,  is  applied 
system-wide.  As  far  as  a  user  is  concerned,  it  can 
be  parameterized  for  certain  well  known  routinq 
techniques  which  will  be  embedded.  In  the  current 
MICROSS  system,  two  variations  of  fixed  routing  have 
already  been  implemented.  The  flooding  as  well  as  an 
adaptive  routing  technique  is  being  implemented. 

Communication  protocols:  these  are  also  parameterized. 
Each  protocol  is  simulated  by  a  fixed  delay  time 
depending  on  the  complexity  involved.  The  protocols 
covered  are  hop-to-hop  HDLC  protocol,  BISYNCH,  X25 
and  a  simple  handshake.  User  defined  protocols  can 
be  incorporated  with  minor  changes,  as  long  as  the 
store-and-forward  aspect  of  the  DCS  is  obeyed. 


D. 


Application  Data 


Application  data  is  modelled  in  terms  of  the  frequency 
of  external  message  input  at  each  node  and  a  traffic 
matrix  indicating  the  traffic  flow  requirements  between 
node  pairs.  The  former  is  presented  in  terms  of 
message  inter  arrival  times  and  the  probability 
distribution  function.  The  latter  is  presented  in 
terms  of  a  probability  matrix.  Optionally,  actual 
traffic  can  be  used  to  feed  to  the  simulated  system. 
This  approach  is  most  helpful  in  testing  the  model 
against  an  existing  counterpart. 

The  traffic  can  also  be  grouped  into  a  number  of 
priority  classes. 

At  various  points  throughout  the  definition  phase  the  user  is 
given  the  opportunity  to  store  his  specific  data  on  files. 

This  aids  in  the  construction  of  more  exact  system  models. 


3.2.3  Simulation  Subsystem 

MICROSS  simulation  concept 

The  simulation  subsystem  utilizes  every  structure  defined  and/or 
reset  in  the  definition  subsystem.  The  software  modularity  is 
attempted  by  functional  partitioning  so  that  easy  access  by  the 
user  is  possible.  There  are  two  disjoint  but  closely  related 
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sets  of  functions  at  higher  level.  The  event  (TX)  scheduling 
in  time  and  priority,  activation  of  a  particular  TX  at  a  point 
in  time  (current  time),  advancement  of  global  time  as  well  as 
updating  the  status  of  the  facilities  at  that  instant  of 
current  time  are  all  carried  out  in  one  of  these  disjoint  sets 
of  subroutines  (using  FORTRAN  language  terminology) .  The  other 
set  of  subroutines  handles  the  facility  seizure  upon  activation 
of  a  particular  TX.  The  seizure  can  take  place  because  of  one 
of  the  following  functions: 

1.  Arrival  of  an  external  TX  at  the  host  computer. 

2.  Local  transmission  attempt  between  the  host  and 
the  local  communication  node. 

3.  Receipt  of  local  external  messages  at  the 
communication  node. 

4.  Receipt  of  remote  traffic  at  the  communication 
node . 

5.  Data  transmission  requests  between  two 
communication  nodes. 

6.  Scheduling  the  amount  of  external  traffic. 

The  model  provides  a  degree  of  interaction  between  some  of 
the  functions.  For  example,  data  transmission  request  at  one 
side  of  a  pair  triggers  the  other  side  for  reception. 

The  last  item,  i.e.  6,  does  not  involve  seizure  of  a 
facility.  The  pattern  of  scheduling  at  a  node  depends  on  the 
application  data  parameters  fed  in  during  system  initialization. 
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Arrival  of  a  message  is  scheduled  but  is  not  guaranteed  to 
be  handled  at  the  host  right  away  unless  the  host  is  free. 

The  first  five  functions  may  require  queuing  at  one  facility 
and  dequeuing  at  the  other  depending  on  the  success  of  action 
that  has  taken  place,  and  also  the  protocols  involved.  For 
example  if  the  protocol  chosen  is  an  HDLC  utilizing  hop-to-hop 
with  piggy-back  acknowledgement  then  an  arriving  message  is 
queued  as  soon  as  transmission  is  complete.  Its  copy  in  the 
sending  node  is  not  removed  from  the  queue  unless  its  receipt 
is  acknowledged. 

The  acknowledgement,  queuing  and  dequeuing  operations  are 
performed  during  the  receive  action  of  the  host  and/or  the 
communication  node. 

Timing  of  each  operation  is  necessary  to  affect  the  seizure 
duration  of  the  facility  in  question.  The  smallest  time 
increment  corresponds  to  shortest  independent  process 
(function).  For  example,  an  attempt  to  transmit  by  a  sender 
may  be  aborted  because  of  the  busy  state  of  the  receiver. 
However,  this  attempt  itself  is  an  executable  process  and  is 
timed  according  to  the  number  of  instructions  and  speed  of 
the  processor  performing  it.  Thus  the  attempted  TX  is 
rescheduled  to  be  ready  for  re-attempt  at  a  later  time. 


Figure  2.1  shows  two  nodes  as  they  are  simulated  by  MICROSS, 
which  appears  as  a  network  of  queues.  The  host  contains  one 
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output  queue  leading  to  the  local  communication  node  via  a 
link.  Communication  node  possesses  m  queues  leading  to  m 
other  communication  nodes  and  another  queue  leading  to  the 
local  host  part. 

The  details  of  the  data  structure  used  to  represent  the 
facilities  are  summarized  in  the  next  section. 

MICROSS  basic  data  structure 

The  MICROSS  data  structure  can  be  grouped  into  four  main 
classes  which  are  closely  interlinked: 

1.  Physical  components. 

2.  Status  of  physical  components. 

3.  Application  traffic  and  routing. 

4.  Book-keeping  constructs. 

Figure  3.3  displays  the  presentation  of  the  physical 
components  and  the  related  statistics  ,  for  all  except 
queues  the  specification  data  and  status  data  are  given  in 
a  simple  array  type  data  structure.  Queue  representation 
requires  higher  sophistication.  The  representation  of  an 
output  queue  is  a  linked  list  structure.  The  status 

data  is  kept  in  a  separate  array.  For  modularity  reasons 
each  queue  is  also  associated  on  a  separate  data  area  to 
store  the  actual  messages. 


The  index  of  a  message  in  the  data  area  is  kept  in  the  right 
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1. 

Processor  id  number. 

1 . 

Processor  id. 

2. 

Instruction  execution  speed (KIPS). 

2. 

Execution  speed  (KIPS) 

3. 

Memory  size  (KBYTE). 

3. 

Number  of  output  ports. 

4. 

Total  external  input  in  number  of 

4. 

Number  of  input  ports. 

transactions* 

5. 

Memory  size  (KBYTE) 

5. 

Total  number  of  TXs  received* 

6. 

Total  number  of  TX  receive 

6. 

Mean  inter-arrival  time  of 

7. 

Total  number  of  TX  sent* 

external  input. 

8. 

(unused) . 

7. 

Total  busy  time* 

9. 

Total  busy  time* 

8. 

Status  (1-busy,  0-free) 

10. 

Status  (0-f ree-l-busy) 

9. 

Time  scheduled  to 

11. 

Time  scheduled  to  be  free. 

10. 

Total  time  the  arriving  TSx  spent 
in  the  system* 

12. 

Protocol  handling  time 

11 . 

Protocol  handling  time 

(a) 

LP  Representation 

(b) 

CP  Representation 

1. 

Link  id. 

1. 

Queue  id . 

2. 

Transmission  capacity  (K  bits/sec) 

2. 

Queue  length. 

3. 

Total  busy  time* 

3. 

Current  content 

4. 

Total  number  of  TXs  handled. 

4. 

Total  occupancy  time* 

5. 

Status  (0-free,  1-busy). 

5. 

Total  number  TX  departed* 

6. 

Time  scheduled. 

6. 

Status  (empty,  full) 

7. 

Transmission  time  (us ) /b i t 

7. 

Send  no.  of  last  TX  sent. 

(*) 

Statistics 

8. 

Maximum  content 

9. 

Reserved 

(c) 

L-Link  representation 

(d) 

Output  Queue  status 

1 . 

TX  id. 

1. 

Data  send  seq.  no. 

2. 

Current  time 

2. 

ack  send  seq.  no. 

3. 

Index  of  TX  in  the  data  area 

3. 

data  receive  seq.  no. 

4. 

Front  pointer 

4. 

ack  receive  seq.  no. 

3. 

6. 

TX  send  sequence  no. 

(unused) 

3. 

validity  flag 

(e) 

An  entry  of  output  Queue 

(f) 

Acknowledgement  tab 

Figure  3.3 


BASIC  DATA  STRUCTURE  LAYOUTS 
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queue.  Therefore  no  search  is  necessary  for  retrieval. 

Further  data  structures  are  added  depending  on  the  protocols 
employed.  For  example,  for  HDLC  protocol,  utilizing  piggy¬ 
back  window  mechanism  for  acknowledgement  another  structure 
is  employed  to  record  the  relevant  data  for  each  queue. 

The  application  data  structure  consists  of  an  nxn  traffic 
matrix  where  each  entry  Tij  indicates  the  proportion  of  the 
traffic  sourced  at  i  and  destined  for  j. 

The  routing  constructs  depend  on  the  specific  technique 
implemented.  For  fixed  touting  technique  an  nxn  routing 
matrix  (R)  where  indicates  next  node  after  i  on  the 

path  to  j  is  the  main  structure.  As  support  constructs,  we 
have  connection  table  (ICONT)  where  each  row  is  a  set  of 
adjacent  nodes  of  the  corresponding  node,  and  weight  table 
w  where  each  Wj_j  indicates  the  minimum  distance  between 
node  i  and  j  in  terms  of  weight  associated  to  each  link  on 
the  path  (i , j ) . 

Two  main  book-keeping  data  constructs  (LISTX,  IACTX)  are 
employed  to  hold  the  list  of  scheduled  but  inactive 
transactions,  and  corresponding  active  transactions. 

The  TXs  pingpong  between  these  two  lists  until  terminated, 
after  which  they  are  removed  from  either  of  the  lists  and 
also  from  the  system  altogether. 
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Both  of  these  constructs  are  represented  by  linked  lists 
in  which  TXs  are  in  time  and  priority  order.  The  TX 
information  carried  in  these  lists  are:- 


1 .  Source  id  of  the  TX 

2.  destination  id 

3.  TX  id 

4.  time  the  TX  is  scheduled  at 

5.  the  action  to  take  place  next 

6.  priority 

7.  link  area 

8.  output  queue  id 

9.  current  node  id. 


The  TXs  are  added  or  removed  from  the  link  lists  using  the 
avail  (garbage  collection)  lists. 

The  data  structure  allows  the  gathering  of  statistics  on  the 
utilization  of  the  system  as  well  as  the  individual  physical 
and  logical  constructs. 

3.2.4  Reporting  Subsystem 

Reporting  is  naturally  referred  to  only  after  the  completion 
of  simulation  for  a  non-zero  increment  of  time.  Once  it  is 


entered  the  level  of  reporting  details  is  chosen  by  the  user. 
The  small  flow  diagram  in  Figure  3.4  shows  the  reporting 
options  available  in  MICROSS. 
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A  short  report  displays  the  main  system  description  data 
as  well  as  the  results  concerning  the  throughput  and  average 
delay  time  experienced.  Figure  3.5  presents  such  a  report 
for  a  16  node  cross-connected  regular  and  homogeneous  DCS- 
In  Figure  3.5,  each  row  of  connection  tables  shows  the  set 
of  adjacent  nodes. 

The  long  report  option  gives  details  of  External  Input,  Node 
Content,  Delivered,  Received  and  Unacknowledged  Messages, 
Response  Times  and  Utilisation  of  Host  and  Communication 
Processors,  Output  Queues  and  Communication  Links  separately. 
A  full  set  of  reports  for  the  16  node  system  is  given  in 
Figure  3.6. 

The  graphics  report  produces  four  graphs  showing 

1 .  Response  Time  statistics 

2.  Nodal  Statistics 

3.  Utilisation  Statistics 

4.  Output  Queue  Status. 

Examples  (again  for  the  16  node  system)  are  given  in 
Figure  3 . 7  (a-d) . 

Notice  that  the  Output  Queue  Status  report  is  essentially 
identical  to  the  System  Snapshot  option  in  MONITOR:  in  both 
cases  an  arrow  represents  a  link  in  the  direction  shown  and 
vertical  bars  behind  the  arrow  represent  messages  in  the 
relevant  output  queue. 


Cl-OIT  UTt-.UI  ATTCt.  REPORT 
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The  graphics  output  may  be  sent  to  a  plotting  device  using 
reporting  Option  4,  and  Option  5  produces  a  data  file 
containing  both  the  short  and  long  reports. 

After  generating  reports  the  user  is  returned  to  the 
MONITOR  where  he  may  simulate  again  after  inputting  an 
increment  to  the  simulation  time  or  he  may  choose  one  of 
the  other  options. 

The  simulation  feedback  cycle  as  shown  in  Figure  3.8  can 
continue  until  the  user  is  satisfied  with  the  networking 
characteristics  of  the  DCS  regarding  topology,  routing 
technique,  communication  protocol,  and  application  traffic. 


4.  GRAPHICS  FACILITIES  IN  MICROSS 

Graphics  aid  in  MICROSS  is  utilized  at  3  levels: 

1 .  Definition 

2.  Simulation 

3.  Reporting 

Graphics  use  in  reporting  is  discussed  in  section  3.3  as  a 
part  of  basic  MICROSS  system,  and  is  used  to  display  the 
statistics  in  classical  sense  in  the  form  of  histograms, 


curves,  etc. 
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'  .  i  Graphics  application  to  system  definition 

Graphics  aid  in  definition  subsystem  is  limited  to  the  node 
level  topological  layouts,  although  technically  there  is 
no  restriction  on  graphical  details  such  as  separation  of 
local  host  from  the  communication  node  and  processor  from 
the  memory.  These  details,  especially  in  a  densely  connected 
environment  lose  their  relevance  and  are  therefore  left  out 
unless  a  multi-processor  structure  is  the  subject. 

The  topological  patterns  at  a  node  level  can  be  formed  from 
the  connection  table,  only  if  the  physical  coordinates  of 
each  node  on  the  screen  were  also  known.  For  regular 
patterns  this  data  can  be  generated  given  the  screen 
particulars,  and  the  connection  table  which  can  also  be  found 
under  the  program  control  triggered  by  a  parameter  indicating 
a  specific  pattern. 

For  irregular  structures  however,  the  coordinates  may  be. 
specified  interactively  under  the  cursor  and  program  control. 
In  this  case  the  coordinates  need  to  be  saved  for 
regeneration  and/or  snapshot  system  state  display  purposes. 

Under  the  program  control  the  cursor  input  is  parameterized 
to  indicate  a  particular  drawing  function  such  as  a  circle 
to  mean  a  node  and  a  line  in  between  the  two  circles  to  mean 
a  link.  The  online  deletions  and  additions  can  also  take 
place.  In  all  cases  it  is  the  topology  information  that  is 
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needed  to  be  saved  for  simulation  and  for  later  display  purposes. 

In  addition  networks  can  be  overlaid  on  the  map  of  a  geographi¬ 
cal  area  selected  from  a  small  available  set  or  from  a  user's 
own  data  file. 

4.2  Graphics  aid  at  simulation  level 

At  the  simulation  level  graphics  can  be  used  to  express  the 
system  state  in  a  concise  manner.  In  MICROSS  this  is  applied 
to  the  current  queue  content  at  each  node  in  the  form  of  a 
network  of  queues  as  it  is  illustrated  in  Figure  3.7. 

The  simulation  and  the  graphics  representation  (or  snapshot) 
are  incorporated  in  MONITOR  section  as  shown  in  Figure  3.2 
where  for  each  snapshot  the  incremental  simulation  time  is 
added  on  to  the  global  time.  The  simulation  can  then  be 
stopped  at  a  point  in  time  after  the  exhaustion  of  the  last 
time  increment. 

In  Figure  3.7  the  circles  represent  the  Communication 
Processor  and  the  square  the  corresponding  Local  Processor; 
the  node  id  is  written  inside  the  square.  A  straight  line 
represents  a  link  between  two  nodes  and  an  arrow  shows  the 
direction  of  the  link  (bidirectional  links  have  two  arrows) . 

The  straight  lines  behind  an  arrow  represent  messages  waiting 
in  that  output  queue.  Similarly  the  arrows  and  lines  inside 
the  Communication  Processor  represent  the  internal  links 
between  Host  and  Communication  Processor. 
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5.  PERFORMANCE  EVALUATION  OF  INTERCONNECTION  TOPOLOGIES 

5.1  Experiment  Specifications 

A  study  has  been  undertaken  to  evaluate  the  performance  of 
a  wide  variety  on  networks  under  the  same  operational 
conditions  so  as  to  determine  the  effect  of  topology  on 
performance.  This  work  also  demonstrates  the  applicability 
of  MICROSS  to  a  relatively  large  class  of  network  structures. 

The  size  of  networks  chosen  range  from  9  to  1 6  nodes.  This 
was  due  to  our  desire  to  study  relatively  large  numbers  of 
networks  within  the  computational  constraints  that  we  have  to 
operate  (DEC  10  with  large  numbers  of  other  users).  Clearly 
MICROSS  is  able  to  handle  much  larger  networks  and  interconnec¬ 
tion  patterns  other  than  those  given  here. 

A  wide  variety  of  9  and  16  nodes  'regular'  and  'standard' 
networks  were  investigated  together  with  a  few  other  networks 
of  interest.  Due  to  time  and  space  limitation,  only  a  subset 
of  possible  interconnection  topologies  is  presented  here. 

One  of  the  main  performance  characteristics  of  a  network  is 
the  time  taken  for  a  message  to  be  delivered  to  its  destina¬ 
tion  since  its  introduction  to  the  system  called  the  Response 
Time.  It  is  normally  required  that  this  time  is  as  short  as 
possible;  sometimes  it  is  essential  that  response  time  is 
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EXTERNAL  TRAFFIC  DISTRIBUTION 
MEAN  INTER  ARRIVAL  TIME 
DISTRIBUTION 
MESSAGE  LENGTH 
DISTRIBUTION 

INTERNAL  MESSAGE  DISTRIBUTION 

PRIORITY  CLASSES 

PROTOCOL 

HARDWARE  CLASSES 
LP  SPEEDS 
CP  SPEEDS 
LP  MEMORY  SIZES 
CP  MEMORY  SIZES 
LP-CP  LINE  SPEEDS 
CP-CP  LINE  SPEEDS 
OUTPUT  QUEUE  LENGTHS 
BUFFER  SIZE 
ROUTING 

SHORTEST  PATHS 
SIMULATION  TIME 


UNIFORM 

2000  microseconds 

UNIFORM 

16  UNITS 

FIXED 

UNIFORM 

NONE 

HANDSHAKE 
NONE 
100  Ki/s 
100  Ki/s 
10  K  UNITS 
10  K  UNITS 
10  K/s  UNITS/sec 
10  K/s  UNITS/sec 
8 
4 

CFIXED 

LOAD  BALANCED 
0 . 1  sec 


Table  5 .  1 


DEFAULT  CONDITIONS 
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less  than  a  critical  maximum  allowable  limit.  Thus,  it  is 
important  to  know  the  profile  of  Response  Times  under  varying 
traffic  conditions.  Average  Repsonse  Time,  Maximum  Response 
Time  and  Response  Time  distribution  are  typical  characteris¬ 
tics  of  interest. 

In  this  study,  we  concentrated  on  the  comparison  of  network 
topologies,  all  operating  under  the  same  traffic  conditions. 
The  set  of  parameters  used  for  these  experiments,  called 
default  conditions  are  given  in  Table  5.1.  As  seen  in  the 
table,  the  message  generation  is  assumed  to  be  uniform  with 
mean  inter  arrival  time  of  2000  microseconds  at  each  node. 
Message  inter  arrival  distribution  is  uniform.  Each  message 
is  assumed  to  be  a  fixed  16  units  long.  When  a  message  is 
generated  at  a  node,  it  is  destined  to  another  node  with 
equal  probability.  There  are  no  priority  classes  in  the 
messages.  All  nodes  contain  the  same  type  of  hardware.  A 
"handshake"  protocol  is  used  for  message  acknowledgement. 

The  routing  is  based  upon  the  load  balanced  algorithm  which 
finds  the  shortest  paths  while  trying  to  distribute  the  load 
evenly  when  there  are  more  than  one?  minimum  length  paths 
between  the  nodes  [19].  Table  5.1  lists  other  characteristics 
of  the  node  computers  and  interconnnecting  lines. 

5.2  Experimental  Results 

The  results  of  the  experiments  are  given  as  a  short  report 
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together  with  a  representation  of  the  network  topology  in 
Appendix  I  (Figures  1  to  33)  for  each  system  and  a  summary 
of  the  main  features  as  tables  5.2  and  5.3.  In  the  tables 
the  results  are  divided  into  two  groups  for  9  and  10  node 
systems  and  for  15  and  16  node  systems.  In  each  group  the 
networks  are  ranked  in  increasing  number  of  output  links. 

The  tables  also  list  the  Average  Response  Time  and  the 
Average  Path  Length  (number  of  inter  node  links  traversed 
by  a  transaction) . 

An  example  of  a  complete  set  of  results  for  a  16  node 
'Regular*  Cross  Connected  network  is  given  as  Figures  3.5  to 
3.7.  For  the  majority  of  experiments  the  package  default 
conditions  were  used,  and  these  are  outlined  in  Table  5.1. 
However,  for  the  'fixed  path  length1  experiments  the  distri¬ 
bution  of  destinations  was  adjusted  so  that  all  transactions 
had  a  known  fixed  path  length  of  1  to  4  (Experiment  26). 

The  results  of  the  experiment  have  also  been  plotted  on  two 
graphs  of: 

1.  Average  Response  Time  vs.  Total  Output  Links 

(Figure  5.1) 

2.  Average  Response  Time  vs.  Average  Path  Length 

(Figure  5.2) 


We  find  a  good  correlation  between  Response  Time  and  Path 


AVERAGE  RESPONSE  TIMES  (  milliseconds 


FIGURE  S . ; 
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9  and  10  NODE  NETWORKS 


NETWORK 

NUMBER 

TOPOLOGY 

RESPONSE 

PATH 

TOTAL  OUTPUT 

(Fig.  No. 
in  Appx.  1) 

OF  NODES 

TIME 

(msec) 

LENGTH 

(average) 

LINKS 

1 

9 

RING  UNIDIRECTIONAL 

24.2 

4.50 

9 

2 

9 

LINE 

20.7 

3.33 

16 

3 

9 

BINARY  TREE 

18.0 

2.67 

16 

4 

9 

STAR 

14.5 

1.79 

16 

5 

9 

RING  BIDIRECTIONAL 

17.7 

2.50 

18 

6 

9 

NET  HEX  CELLS  3x3 

17.5 

2.47 

18 

7 

9 

SELCHUCK 

15.5 

2.00 

18 

8 

9 

NET  SQUARE  CELLS 

15.5 

2.00 

24 

9 

9 

HEX  CELLS  REGULAR 

14.6 

1.81 

27 

10 

9 

HYPERCUBE  (INCOMPLETE) 

15.3 

1.83 

30 

11 

10 

PETERSEN  GRAPH 

14.2 

1  .67 

30 

12 

9 

NET  TRIANGULAR  CELLS 

14.1 

1.67 

32 

13 

9 

CROSS  CONNECTED  (X  CON) 

13.4 

1.50 

36 

14 

9 

X  CON  +  ALT  1st.  DIAG 

12.5 

1.25 

54 

15 

9 

X  CON  +  PARALLEL 

12.3 

1.25 

54 

16 

9 

X  CON  +  H  BONE 

12.5 

1.25 

54 

17 

9 

X  CON  +  BOTH 

11.4 

1 .00 

72 

TABLE  5.2 
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15  and  16  NODE  NETWORKS 


NETWORK 

NUMBER 

TOPOLOGY 

RESPONSE 

PATH 

TOTAL  OUTPUT 

iF  ig.  No. 

in  Appx. 

OF  NODES 

I) 

TIME 

(msec) 

LENGTH 

(average) 

LINKS 

10 

16 

RING 

UNIDIRECTIONAL 

40.4 

8.00 

16 

19 

16 

RING 

BIDIRECTIONAL 

24.3 

4.23 

32 

20 

16 

SELCHUCK 

19.5 

2.93 

32 

21 

15 

BINARY  TREE 

21.4 

3.51 

36 

22 

16 

NET  HEX  CELLS  4x4 

20.5 

3.21 

’  36 

23 

16 

NET  SQUARE  CELLS 

17.8 

2.67 

48 

24 

16 

HEX  CELLS  (REGUALR) 

17.2 

2.40 

48 

25 

15 

WEB  NESTED  PENTAGONS 

16.5 

2.11 

50 

26 

16 

CROSS 

CONNECTED  (X  CON) 

16.0 

2.13 

64 

26 

16 

X  CON 

FIXED  PATH  LGTH 

11.5 

1 

64 

26 

16 

X  CON 

FIXED  PATH  LGTH 

15.5 

2 

64 

26 

16 

X  CON 

FIXED  PATH  LGTH 

19.5 

3 

64 

26 

16 

X  CON 

FIXED  PATH  LGTH 

23.5 

4 

64 

27 

16 

HYPERCUBE 

16.0 

2.07 

64 

28 

16 

NET  TRIANG  CELLS  4x4 

16.3 

2.23 

66 

29 

16 

X  CON 

+  BOTH  2nd 

14.2 

1.67 

80 

30 

16 

X  CON 

+  ALT  1st 

13.9 

1.60 

96 

31 

16 

X  CON 

+  PARALLEL  1st 

13.8 

1.60 

96 

32 

16 

X  CON 

+  H  BONE  1st 

14.  1 

1.67 

96 

33 

16 

X  CON 

+  BOTH  1st 

13.4 

1.47 

128 

TABLE  5 . 3 
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Length  but  not  such  good  correlation  between  Response  Time 
and  Total  Links  although  a  general  trend  is  evident. 

5.3  Delay  time  calculations 

Consider  a  message  generated  at  node  I  whose  destination  is 
node  J,  and  let  this  transfer  take  place  via  a  path  of  length 
N.  Then  the  total  unavoidable  delay,  (i.e.  ignoring  queuing) 
can  be  broken  down  into  groups: 

1.  Delays  at  Host  Processor  I  DH(I) 

2.  Inter  Processor  Delays 


i) 

Host  to  Communication 

at 

node  I 

DHC ( I ) 

ii) 

Communication  to  Host 

at 

node  J 

DCH (J) 

3.  Communication  Delays 

i)  Communication  Processor  at  node  K, 

where  K  is  on  the  path  from  I  to  J 
(including  I  and  J)  DC(K) 

ii)  Communication  link  from  node  K  to 

node  L  DL(K,L) 

As  we  have  taken  default  conditions  where  all  processors  and 
lines  are  identical  the  total  delay  can  be  simplified  as: 

Total  delay  =  2*DH  +  2*DHC  +  (N+1)*DC  +  N*DL  (5.1) 
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It  can  be  seen  that  the  Total  Delay  depends  on  path  length 
linearly  and  the  shape  is  given  by 

DC  +  DL. 

For  the  default  conditions  used  the  packet  length  is  16  units 
to  which  an  overhead  of  another  16  units  is  added  and  the  line 
speeds  are  1 0K  units/sec.  Therefore  we  would  expect  DL  to  be 
32/10  msec  =  3.2  msec. 

As  presently  implemented  the  Communication  Processor  delay  DC 
is  manifested  as  a  delay  of  0.2  msec  under  default  conditions. 
Thus  we  would  expect  the  slope  of  the  Response  Time  vs.  Path 
Length  curve  to  be  at  least  3.4  msec/unit. 

The  slope  determined  experimentally  from  Figure  5.2  is  4.0 
msec /unit.  The  extra  0.6  msec/unit  can  be  accounted  as  due 
to  queuing  delays. 

The  fixed  length  experiments  (Experiment  26)  confirm  this. 

In  each  of  these  experiments  the  path  length  of  all  messages 
was  kept  constant.  The  four  sets  of  results  refer  to  four 
fixed  path  length  values  of  1  to  4.  In  these  cases  the  average 
response  time  corresponds  to  the  Total  Delay  in  (5.1). 

As  an  extension  of  this  work  a  further  series  of  experiments 
using  different  message  lengths  will  be  performed.  Also  a 
series  under  heavier  traffic  rates  to  investigate  delays  due 
to  congestion  will  be  undertaken. 
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Appendix  I 


Experimental  Results  for 
some  9  &  10  and  15  &  16  node 
Interconnection  Topologies 


Figure 
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Figure  29 
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Figure  31 
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Figure  33 


